01 Data Ingestion ¶

Machine learning tasks are typically data heavy, requiring labelled data for supervised learning or unlabelled data for unsupervised learning. In Python, data is typically stored in memory as NumPy arrays at some level, but in most cases you can use higher-level containers built on top of NumPy that are more convenient for tabular data ( Pandas ), multidimensional gridded data ( xarray ), or out-of-core and distributed data ( Dask ).

Each of these libraries allows reading local data in a variety of formats. In many cases the required datasets are large and stored on remote servers, so we will show how to use the Intake library to fetch remote datasets efficiently, including built-in caching to avoid unncessary downloads when the files are available locally.

To ensure that you understand the properties of your data and how it gets transformed at each step in the workflow, we will use exploratory visualization tools as soon as the data is available and at every subsequent step.

Once you have loaded your data, you will typically need to reshape it appropriately before it can be fed into a machine learning pipeline. Those steps will be detailed in the next tutorial: Alignment and Preprocessing .

Inline loading ¶

We'll start with the simple case of loading small local datasets, such as a .csv file for Pandas:

We can inspect the first several lines of the file using .head , or a random set of rows using .sample(n)

To get a better sense of how this dataframe is set up, we can look at .info()

To use methods like pd.read_csv , the data all needs to be on the local filesystem (or on one of the limited remote specification formats supported by Pandas, such as S3). We could of course put in various commands here to fetch a file explicitly from a remote server, but the notebook would then very quickly get complex and unreadable.

Instead, for larger datasets, we can automate those steps using intake so that remote and local data can be treated similarly.

To get better insight into the data without loading it all in just yet, we can inspect the data using .to_dask()

To get a full pandas.DataFrame object, use .read() to load in all the data.

NOTE: There are different items in these two info views which reflect what is knowable before and after we read all the data. For instance, it is not possible to know the shape of the whole dataset before it is loaded.

Loading multiple files ¶

In addition to allowing partitioned reading of files, intake lets the user load and concatenate data across multiple files in one command

NOTE: The length of the dataframe has increased now that we are loading multiple sets of training data.

This can be more simply expressed as:

Sometimes, there is data encoded in a file name or path that causes concatenated data to lose some important context. In this example, we lose the information about which version of landsat the training was done on. To keep track of that information, we can use a python format string to specify our path and declare a new field on our data. That field will get populated based on its value in the path.

Using Catalogs ¶

For more complicated setups, we use the file catalog.yml to declare how the data should be loaded. The catalog lays out how the data should be loaded, defines some metadata, and specifies any patterns in the file path that should be included in the data. Here is an example of a catalog entry:

sources:
  l5:
    description: Images contain Landsat 5 Surface Reflectance Level-2 Science Product.
    driver: rasterio
    cache:
      - argkey: urlpath
        regex: 'earth-data/landsat'
        type: file
    args:
      urlpath: 's3://earth-data/landsat/LT05_L1TP_042033_19881022_20161001_01_T1_sr_band{band:d}.tif'
      chunks:
        band: 1
        x: 256
        y: 256
      concat_dim: band
      storage_options: {'anon': True}

The urlpath can be a path to a file, list of files, or a path with glob notation. Alternatively the path can be written as a python style format_string . In the case where the urlpath is a format string, the fields specified in that string will be parsed from the filenames and returned in the data.

NOTE: If you don't have the data cached yet, then the next cell will take several minutes.

The data has not yet been loaded so we don't have access to the actual data values yet, but we do have access to coordinates and metadata. Next we will read in the data:

Visualizing the data ¶

To get a quick sense of the data, we can plot it using hvPlot , which provides interactive plotting commands for Intake, Pandas, XArray, Dask, and GeoPandas. We'll look more closely at hvPlot and its options in later tutorials.

We can quickly generate a plot of each of the landsat bands:

This same plot can be declared in the catalog for ease of use and to point users to helpful ways to visualize data. Here is the relevant part of catalog.yml :

metadata:
  plots:
    band_image:
      kind: 'image'
      x: 'x'
      y: 'y'
      groupby: 'band'
      datashade: True
      width: 400
      dynamic: False

We can achieve the same output using the underlying XArray DataArray itself. When using the DataArray, we can do some pre-processing such as filtering out missing values.

We can plot this filtered array to get rid of the background artifact seen above.

Since xarray objects store raster information in a predictable way, we can even project the image onto lat lon coordinates using the geo key word argument.

Accessing the data ¶

Machine Learning pipelines such as scikit-learn accept Numpy arrays as input. These arrays are accessible in DataArray objects on the values attribute.

We can use tab completion to explore what other information is stored on our xarray.DataArray object. We can use tab completion to explore attributed and methods available on our object.

Next: ¶

Now that you have loaded your data, you will typically need to reshape it appropriately before it can be fed into a machine-learning pipeline. These steps are detailed in the next tutorial: Alignment and Preprocessing .